MOOCs Meet Measurement Theory: A Topic-Modelling Approach

This paper adapts topic models to the psychometric testing of MOOC students based on their online forum postings. Measurement theory from education and psychology provides statistical models for quantifying a person’s attainment of intangible attributes such as attitudes, abilities or intelligence. Such models infer latent skill levels by relating them to individuals’ observed responses on a series of items such as quiz questions. The set of items can be used to measure a latent skill if individuals’ responses on them conform to a Guttman scale. Such well-scaled items differentiate between individuals and inferred levels span the entire range from most basic to the advanced. In practice, education researchers manually devise items (quiz questions) while optimising well-scaled conformance. Due to the costly nature and expert requirements of this process, psychometric testing has found limited use in everyday teaching. We aim to develop usable measurement models for highly-instrumented MOOC delivery platforms, by using participation in automatically-extracted online forum topics as items. The challenge is to formalise the Guttman scale educational constraint and incorporate it into topic models. To favour topics that automatically conform to a Guttman scale, we introduce a novel regularisation into non-negative matrix factorisation-based topic modelling. We demonstrate the suitability of our approach with both quantitative experiments on three Coursera MOOCs, and with a qualitative survey of topic interpretability on two MOOCs by domain expert interviews. Introduction Massive Open Online Courses (MOOCs) have recently been the subject of a number of studies within disciplines as varied as education, psychology and computer science (Ramesh et al. 2014c; Anderson et al. 2014; Kizilcec, Piech, and Schneider 2013; Dıez et al. 2013; Milligan 2015). With few studies taking a truly cross-disciplinary approach, this paper is the first to marry topic modelling with measurement theory from education and psychology. Measurement in education and psychology is the process of assigning a number to an attribute of an individual in such Copyright c © 2016, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. a way that individuals can be compared to one another (Pedhazur and Schmelkin 1991). These attributes are often intangible such as attitudes, abilities or intelligence. Since the attribute to be measured is not directly observable, a set of items is often devised manually and individuals’ responses on the items are collected. Based on a modelled correspondence with observed item responses, latent attribute levels of a cohort can be inferred. This process is called scaling (De Ayala 2013). A Guttman scale (Guttman 1950) is one which induces a total ordering on items—an individual who successfully answers/agrees with a particular item also answers/agrees with items of lower rank-order. Table 1 depicts an example Guttman scale measuring mathematical ability (Abdi 2010), where the items are ordered in increasing latent difficulty, from Counting to Division. Here the total score corresponds to the persons’ latent ability: the greater the higher. Table 1: An example of a perfect Guttman scale measuring mathematical ability(Abdi 2010) , where 1 means the person has mastered the item and 0 for not. Person 5 who has mastered the most difficult item Division, is expected to have mastered all easier items as well. Item 1 Item 2 Item 3 Item 4 Item 5 Total (Counting) (+) (−) (×) (÷) Score Person 1 1 0 0 0 0 1 Person 2 1 1 0 0 0 2 Person 3 1 1 1 0 0 3 Person 4 1 1 1 1 0 4 Person 5 1 1 1 1 1 5 In MOOCs, as in the traditional classroom, we may hypothesise that students possess a latent ability in the subject at hand. For example, in a MOOC on macroeconomics, students are expected to develop knowledge in introductory macroeconomics via videos, quizzes and forums. Students’ latent abilities can be defined, validated and measured using indicators drawn from student responses to activities like interaction with videos, quiz results and forum participation. Unlike the traditional classroom, MOOCs create new challenges and opportunities for measurement through the multiple modes of student interaction online—all monitored at large scale. The education research community is broadly interested in whether and how latent complex patterns of ar X iv :1 51 1. 07 96 1v 1 [ cs .L G ] 2 5 N ov 2 01 5 engagement might evidence the possession of a latent skill, and not just explanatory variables (e.g., visible quizzes and assignments) by themselves (Milligan 2015). This paper focuses on using the content of forum discussion in MOOCs for measurement, which is too timeconsuming to analyse manually but that can provide a predictive indicator of achievement (Beaudoin 2002). We automatically generate items (topics) from unstructured forum data using topic modelling. Our goal is to discover items on which dichotomous (posting on a topic or not) student responses conform to a Guttman scale; where items are interpretable to subject-matter experts who could be teaching such MOOCs. For example, for a MOOC on discrete optimisation, our goal is to automatically discover topics such as How to use platform/python—the easiest which most students contribute to—and How to design and tune simulated annealing and local search—a more difficult topic which only a few students might post on. Such well-scaled items can be used for curriculum design and student assessment. The challenge is to formalise the Guttman scale educational constraint and incorporate it into topic models. We opt to focus on non-negative matrix factorisation (NMF) approaches to topic modelling, as these admit natural integration of the Guttman scale educational constraint. Contributions. The main contributions of this paper are: • A first study of how a machine learning technique, NMFbased topic modelling, can be used for the education research topic of psychometric testing; • A novel regularisation of NMF that incorporates the educational constraint that inferred topics form a Guttman scale; and accompanying training algorithm; • Quantitative experiments on three Coursera MOOCs covering a broad swath of disciplines, establishing statistical effectiveness of our algorithm; and • A carefully designed qualitative survey of experts in two MOOC subjects, which supports the interpretability of our results and suggests their applicability in education. Related Work Various studies have been conducted into MOOCs for tasks such as dropout prediction (Halawa, Greene, and Mitchell 2014; Yang et al. 2013; Ramesh et al. 2014b; Kloft et al. 2014; He et al. 2015), characterising student engagement (Anderson et al. 2014; Kizilcec, Piech, and Schneider 2013; Ramesh et al. 2014b) and peer assessment (Dıez et al. 2013; Piech et al. 2013; Mi and Yeung 2015). Forum discussions in MOOCs have been of interest recently, due to the availability of rich textual data and social behaviour. For example, Wen, Yang, and Rose (2014) use sentiment analysis to monitor students’ trending opinions towards the course and to correlate sentiment with dropouts over time using survival analysis. Yang et al. (2015) predict students’ confusion with learning activities as expressed in the discussion forums using discussion behaviour and clickstream data, and explore the impact of confusion on student dropout. Ramesh et al. (2015) predict sentiment in MOOC forums using hinge-loss Markov random fields. Yang, Adamson, and Rosé (2014) study question recommendation in discussion forums based on matrix factorisation. Gillani et al. (2014) find communities using Bayesian Non-Negative Matrix Factorisation. Despite this variety of works, no machine learning research has explored forum discussions for the purpose of measurement in MOOCs. Topic modelling has been applied in MOOCs for tasks such as understanding key themes in forum discussions (Robinson 2015), predicting student survival (Ramesh et al. 2014a), study partner recommendation (Xu and Yang 2015) and course recommendation (Apaza et al. 2014). However, to our knowledge, no studies have leveraged topic modelling for measurement. More generally, psychometric models have enjoyed only fleeting attention by the machine learning community previously. Preliminaries and Problem Formalisation We choose NMF as the basic approach to discover forum topics due to the interpretability of topics produced, and the extensibility of its optimisation program. We begin with a brief overview of NMF and then define our problem. Non-Negative Matrix Factorisation (NMF) Given a non-negative matrix V ∈ Rm×n and a positive integer k, NMF factorises V into the product of a non-negative matrix W ∈ Rm×k and a non-negative matrix H ∈ Rk×n V ≈WH A commonly-used measure for quantifying the quality of this approximation is the Frobenius norm between V and WH. Thus, NMF involves solving the following optimisation problem, min W,H ‖V −WH‖F s.t. W ≥ 0, H ≥ 0 . (1) The objective function is convex in W and H separately, but not together. Therefore standard optimisers are not expected to find a global optimum. The multiplicative update algorithm (Lee and Seung 2001) is commonly used to find a local optimum, where W and H are updated by a multiplicative factor that depends on the quality of the approximation. Problem Statement We explore the automatic discovery of forum discussion topics for measurement in MOOCs. Our central tenet is that topics can be regarded as useful items for measuring a latent skill, if student responses to these items conform to a Guttman scale, and if the topics are semantically-meaningful to domain experts. As Guttman scale item responses are typically dichotomous, we consider item responses to be whether a student posts on the topic or not. Our goal is to generate a set of meaningful topics that yield a student-topic matrix conforming to the properties of a Guttman scale, e.g., a near-triangular matrix (see Table 1). This process can be cast as optimisation. We apply such well-scaled topics to measure skill attainment—as level of forum participation is known to be predictive of learning outcomes (Beaudoin 2002). Using NMF, a word-student matrix V can be factorised into two non-negative matrices: word-topic matrix W and topic-student matrix H. Our application requires that the topic-student matrix H be a) Binary ensuring the response of a student to a topic is dichotomous; and b) Guttmanscaled ensuring the student responses to topics conform to a Guttman scale. NMF provides an elegant framework for incorporating these educational constraints via adding novel regularisation, as detailed in the next section. A glossary of important symbols used in this paper is given in Table 2. Table 2: Glossary of symbols